Memory optimized dists_add_symmetric #18
Conversation
| dists = dists[ids_row, assign, ids_col] | ||
| return dists | ||
| distances = torch.cdist(TXO_gt_points, TXO_pred_points, | ||
| p=2, compute_mode='donot_use_mm_for_euclid_dist') |
There was a problem hiding this comment.
compute_mode='donot_use_mm_for_euclid_dist' is the important parameter value.
Checked on the debug data instance: for the default mode, it gives ~3% of points got a different closest point id.
Yet the difference is about 1e-4--1e-6, but the performance benefit is rather small.
|
Thanks @KushnirDmytro for this work ! Maybe @ElliotMaitre you could test this to double check ? |
|
I tested it, it works for me. This change allows to run the evaluation on tless dataset, on a RTX-3060 (12Gb). However, on the ycbv dataset, I still have the memory issue with Thank you for your contribution |
|
@ElliotMaitre After your comment, I was puzzled by a reported YCBV dataset problem. Then checked the data:
It feels like you had |
I am proposing PR that fixes the old issues that mention 'CUDA out of memory error' upon running the evaluation script.
I figured out that this issue comes from a single function;
It is
cosypose.lib3d.distances.dist_add_symmetricIt allocates tensors of sizes NxNx3 and NxNx1, where N is the number of points.
Yet the same could be achieved by rewriting the code a little bit.
Alternative solutions are also possible and working(tested).
float16Those approaches are tested but not used in the current proposal. Yet, they could be added later if the issue arises again with larger point clouds or the requirement to run on constrained hardware.
Also, some distance functions from lib3d.symmetric_distances.py file could be optimized, as they compute similar distance functions.
This solution uses <0.25 of the original version's memory:
An experiment was performed on run_cosy_pose_eval.py pipeline.
Evaluated 30 objects from
tless.bopversion of the dataset.The experiment with RTX-2080(8Gb) was not clean because GPU was also used for system GUI runtime.
The scenario with TITAN-X(12Gb) was much cleaner - performed on a headless server.
The old version of the code fails on both setups, while the new one works on both.
The low threshold on use cases could be explained by memory usage for context data and fragmentation. The error is triggered by the requirement to allocate one very large contiguous Tensor. This PR fixes the